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Abstract 

The term Fusion Interface is defined as a class of interface which in- 
tegrally incorporates both virtual and non-virtual concepts and devices 
across the visual, auditory and haptic sensory modalities. A fusion in- 
terface is a multi-sensory virtually-augmented synthetic environment. A 
new facility has been developed within the Human Engineering Division 
of the Armstrong Laboratory dedicated to exploratory development of 
fusion interface concepts. This new facility, the Fusion Interfaces for 
Tactical Environments facility, or FITE, is a specialized flight simula- 
tor enabling efficient concept development through rapid prototyping and 
direct experience of new fusion concepts. The FITE facility also supports 
evaluation of fusion concepts by operational fighter pilots in an air com- 
bat environment. The facility is utilized by a multi-disciplinary design 
team composed of human factors engineers, electronics engineers, com- 
puter scientists, experimental psychologists, and operational pilots. The 
FITE computational architecture is composed of 25 80486-based micro- 
computers operating in real-time. The micro-computers generate out-the- 
window visuals, in-cockpit and head-mounted visuals, localized auditory 
presentations, haptic displays on the stick and rudder pedals, as well as 
executing weapons models, aerodynamic models, and threat models. 


1 Introduction 

Future tactical aircraft will be operating in a much more demanding environ- 
ment than they do today. The lethality of weapons systems continues to in- 
crease. The proliferation of advanced weapon technologies such as directed en- 
ergy weapons, reduced target detectability, and increasing use of passive sensor 
methods will create increased air combat dynamics as well as requiring addi- 
tional time critical decisions to be made by the pilots of future tactical aircraft. 
This will place the tactical aircraft pilot in a more difficult mission environment 
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than he currently experiences. To counter the increasingly complex tactical 
environment, advanced pilot-vehicle interface (PVI) techniques, which enable 
a more efficient use of the pilot’s abilities, are required to be developed and 
employed, thus providing greater situation awareness, enhanced controllability 
of the overall weapon system, increased lethality, and increased survivability. 
These factors, in turn, combine to enhance weapon system effectiveness. 

The cockpit is not the only environment in which advanced interfaces may 
increase the performance of a total system. The link, or interface, between 
man and intelligent machine, may be limiting the productivity and efficiency 
obtainable through automation and machine intelligence [3]. The challenges in- 
volved in linking humans and intelligent machines are increasing due to increased 
computational power, increased availability, increased machine- to-machine com- 
munication capability, increased functional capability through higher level lan- 
guages, and increased memory capacity. 

The PVI can be viewed as a type of human-machine interface, with the ma- 
chine representing the advanced avionics system. This generalization highlights 
an opportunity, that being the opportunity to transition military-sector ad- 
vanced PVI concepts and devices to the commercial-sector, as human- machine 
interface concepts and devices. This paper describes ongoing research and ex- 
ploratory development of multi-sensory virtual interfaces, an advanced PVI 
technology within the Human Engineering Division of the Armstrong Labo- 
ratory. 

2 Multi-sensory virtual interfaces 

A melding, through the interface, between user and system can couple the induc- 
tive capability of the system with the enormous deductive power of the human 
user. Advanced multi-sensory interface concepts and technologies can aid this 
melding process by providing a flexible and adaptive interface medium. 

Multi-sensory virtual-interfaces may enhance weapon system performance 
during tactical engagements by contemporaneously and coherently providing 
display of multiple-sensory channels of information to the fighter pilot and en- 
abling novel control methodologies to be employed through the use of virtual 
devices. In the same way, multi-sensory virtual-interface concepts may provide 
enhancements in other human-machine links by capitalizing on the human’s 
innate ability to integrate, assimilate, and fuse multiple sensory experiences si- 
multaneously. Multi-sensory interfaces may better support the interactions nec- 
essary to fully realize and direct the capability of increasingly powerful avionics 
and weapon systems. 

The application of advanced, multi-sensory interface concepts may best be 
accomplished using a combination of non-virtual (or conventional) and virtual 
control and display devices. The use of a combination of virtual and non- 
virtual devices can create a novel experience for the user, such as in high- 
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fidelity embedded in-flight training, wide field-of-view multi-sensory imagery 
portrayal, or in cockpit portrayals adaptive to the tactical situation and the 
pilot’s condition. This type of experience, in the general sense, has been termed 
in the current literature as virtual reality [10]. The perceptual space created by 
this experience has been termed the virtual environment and more recently, the 
synthetic environment. 


3 Enabling Devices for Virtual Interfaces 

Three sense modalities lend themselves for use as virtual interfaces within tac- 
tical environments. These are vision, audition, and touch. Touch may be more 
clearly defined, in this context, as haptic, involving both cutaneous and kines- 
thetic stimulation. Several technologies are currently being utilized to create 
virtual interfaces within these modalities. Some of these technologies are very 
mature while others are in their infancy. 

The technology to totally emulate naturally occurring environments within 
a virtual environment does not exist at this time. However, devices enabling 
the creation of a limited virtual experience do exist [3]. Helmet mounted dis- 
plays, helmet mounted head, hand, and eye line-of-sight trackers, three dimen- 
sional auditory displays, and tactile stimulation devices have been developed 
and evaluated by several academic, industrial, and military institutions in a 
non-integrated fashion or, context independently, since the mid 1960s [4], [5], 
[12], [14], [13], [11]. Perceptual research which may impact the design, develop- 
ment and application of virtual environment technology in the areas of vision, 
audition, and proprioception is continuing internationally. 

3.1 Virtual Visual Devices 

The virtual visual interface is composed of a visual image generator and a vi- 
sual image portrayal system. The visual image generator can be a computer 
graphics system, an imaging camera, or other similar sensor. Of the three 
modalities, the technology for visual virtual interface application is the most ma- 
ture. Computer graphics, sensor technology, and image portrayal mechanisms 
are important technologies for the creation of robust and high-fidelity virtual 
environments. Image portrayal technologies involved in the transformation of 
electronic-formatted imagery to a visual format include miniature cathode-ray 
tubes, liquid crystal displays, optics, interface electronics. Many of these tech- 
nologies are being considered for incorporation, or are already integrated, into 
aircraft cockpits. These include HUDs, and helmet mounted displays (HMD), 
as well as three-dimensional panel mounted displays. Virtual visual controls 
include helmet mounted head position/attitude trackers and eye line-of-sight 
trackers. A discussion of helmet mounted display technologies can be found in 
a paper by Kocian [6]. 
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3.2 Virtual Auditory Devices 

Virtual auditory displays have been developed which utilize three-dimensional 
auditory localizers combined with audio image sources and stereo headsets. The 
three-dimensional auditory localizer samples the auditory image created by an 
audio image source such as an intercom or computer generated tone source, digi- 
tally filters the signal based on a head-related transfer function that is a function 
of sound location in azimuth, elevation, and range, and results in a stereo audio 
pair. The stereo pair is converted to an analog form and displayed over a stereo 
headset. A discussion of current technology and perceptual research issues con- 
cerning auditory localization, is found in papers by McKinley [7], Wenzel [15] 
and Durlach [2]. 

3.3 Virtual Haptic Devices and Virtual Control 

Haptic displays, which enable the portrayal of virtual cutaneous and kinesthetic 
information, and other virtual control methods, are less developed technologies 
then the visual and auditory technologies, but may be made available for use as 
currently ongoing research makes advances. These include tactile/haptic stim- 
ulation devices, hand and body flexure measurement devices, direct vestibular 
stimulators, direct retinal displays, and directly-coupled brain-actuated control. 
Control loaders, typically utilized in flight simulation to accurately model the 
stick feel of a particular aircraft, also fall into this category but are a ma- 
ture technology. Some of these technologies are discussed by Rheingold [10], 
Meyer [8], and Monkman [9]. 


4 Fusion interfaces 

The term Fusion Interface is utilized within this context as a class of interface 
which utilizes both virtual and non-virtual concepts and devices integrally across 
the visual, auditory and haptic modalities. A fusion interface is a multi-sensory 
virtually-augmented interface. 

Although much is known about human perception and performance [1], the 
creation and evaluation of multi-sensory virtually-augmented interfaces presents 
a design problem in that it is difficult to design or evaluate these interface at- 
tributes without experiencing them within the application context. In many 
cases, what seems to be a valid display or control concept breaks down when 
it is experienced. For this reason, multi-sensory virtual-interface creation, in- 
tegration, and evaluation can best be accomplished within a rapid prototyping 
environment capable of generating the virtually-augmented environment for the 
pilot. The Fusion Interfaces for Tactical Environments (FITE) laboratory, a 
facility within the Crew Systems Directorate of the Armstrong Laboratory, has 
this capability and is currently supporting the exploratory development of multi- 
sensory virtual-interface technologies for tactical cockpits by a multi-disciplinary 
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team composed of operational pilots, human factors engineers, electronics engi- 
neers, computer scientists, and experimental psychologists. 

4.1 Development Process within the FITE 

The products of the FITE laboratory are the multi-sensory virtual-interface 
concepts themselves, the evaluations of the interface concepts, and the identi- 
fication of basic research topics to be pursued in more controlled experimental 
settings. The FITE laboratory operates in three main cycles. These cycles, and 
the resultant products, are depicted in Figure 1. The three cycles are the rapid 
prototyping loop, enabling the rapid creation and initial evaluation of concepts, 
the basic research loop, outputing basic research topics to other facilities and 
inputting results back into the interface design process, and the evaluation loop, 
which is the traditional simulation evaluation loop. 

4.2 FITE Hardware Architecture 

The central component of the FITE laboratory is a F-16 cockpit shell which 
is the focus of the multi- sensory virtual-interface concept development and 
evaluation. The other components of interest in the FITE are its computational 
structure, its virtual and non-virtual visual displays, its virtual auditory display, 
and its haptic display. A block diagram of the hardware architecture is show in 
Figure 2. 

4.2.1 Computational Structure 

The computational structure consists of a distributed collection of 80486-33 
MHz computers configured with 4 Mbytes of memory and a single high-density 
floppy drive. This computer configuration provides real - time performance 
inexpensively. The computers interface in real-time through an off-the-shelf 128 
Byte shared-memory unit manufactured by Bit-3. The computer communica- 
tions through the shared-memory is transparent to the software executing within 
the computers enabling software protocols to be eliminated. Software develop- 
ment is performed on two 80486-33 MHz machines configured with hard-drives 
and off- the- shelf compilers and linkers. Software development and run-time 
execution is performed under the DOS operating system. High-C and Absoft 
FORTRAN are the software languages used and are linked with the Phar- Lap 
DOS extender enabling utilization of extended memory. 

All of the visual displays are generated using Irisvision boards, which are 
an off-the-shelf two-board graphics card set fabricated by Silicon Graphics and 
Pelucid. The Irisvision boards are physically located inside the 80486-33 MHz 
systems. The graphics systems provide several video formats ranging from stan- 
dard television formats (NTSC or PAL) to 1280 by 1024 pixels, non-interlaced, 
at 60 Hz. The FITE facility has standardized on a 1024 by 768 pixel format, 
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interlaced, at 30 Hz . Independent of video format, the Irisvision graphics sys- 
tems provide double-buffering, anti-aliasing, 24-bit z-buffering, Gouraud shad- 
ing. 90,000 triangles/second, and over 14,000 shaded triangles/second. This ca- 
pability supports symbology-only, imagery- only, and symbology-over-imagery 
graphical depictions. Data acquisition, digital-to- analog, and analog- to- digital 
conversion is performed by off-the-shelf data acquisition cards, manufactured 
by Computer Boards Inc, which are incorporated into the 80486-33 MHz com- 
puters. 

The tactical environment, including aerodynamic modeling, weapons mod- 
eling, avionics modeling, as well as digital and manned threat modeling, are 
computed on distributed 80486-33 MHz systems. The processors within the 
FITE cycle in real-time at integer multiples of the visual imagery update rate of 
30 Hz due to synchronization considerations. While most of the processors oper- 
ate at 30 Hz, , the manned threat stations, the out- the- window scene generation, 
and the complex aerodynamic models operate at 15 Hz. 

4.2.2 FITE Visual Displays 

The FITE laboratory incorporates several non-virtual and virtual visual dis- 
plays. The front panel of the F-16 cockpit shell is filled with 6 off-the-shelf 
color LCDs forming a composite large area head-down display. The head- 
down display is virtually-augmented with a see-through HMD, the Tactical 
All- Aspect Helmet Mounted Display (TAAHMD). The TAAHMD provides an 
instantaneous monocular 20° field-of-view wherever the pilot aims his head. 
The TAAHMD is used not only for displaying information, but also for aim- 
ing weapons and pointing avionics. A head-tracker is incorporated into the 
TAAHMD and allows displays to be stabilized to the outside world, to other 
aircraft, or to the cockpit, allowing augmentation of head-down displays. Out- 
the-window visuals are portrayed using 6 off-the-shelf monochrome-white pro- 
jectors focused onto flat screens forming a cube around the F-16 cockpit shell. 
The cube is approximately 3.7m on each edge and the out-the-window projec- 
tions cover an area slightly greater then the frontal hemisphere relative to the 
pilot. A single monochrome-green projector is used to portray HUD imagery. 
Each of the LCDs, the projectors, and the HMD is driven by a graphics card-set 
incorporated into a computer system for a total of 14 systems utilized for visual 
display. 

Three Kaiser Sim-Eye HMDs, which are binocular, monochrome, and in- 
corporate a 60° by 40° field of view, are integrated into flight stations which 
support other participants during real-time simulation. These helmet mounted 
displays are used to portray the out-the-window scene as well as a totally virtual 
visual cockpit for the other participants. 
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4.2.3 FITE Audio Displays 


Audio signals from tone generators, intercoms, and speech synthesizers are 
driven through an auditory localizer which provides spatially-linked auditory 
displays. Tone generation and speech synthesis are performed by Sound Blaster 
Pro cards, manufactured by Creative Labs Inc. The auditory localizer, a Con- 
volvotron, is also off-the-shelf and is manufactured by Crystal River Engineer- 
ing. The auditory localizer can input four audio signals and localize in azimuth, 
elevation, and range. Because the azimuth, elevation, and range provided to 
the localizer is computed within the flight simulation, various localization algo- 
rithms can be employed, ranging from radar warning receiver tones emanating 
from the aircraft producing the radar energy, to using localization to enhance 
non-position derived information, such as tail aspect angle of an attacking air- 
craft. The intercom allows voice communications between individuals within 
the flight environment with individual stations being localized in real-time. 

4.2.4 FITE Haptic Displays 

A McFadden hydraulic control-loader provides real-time-modifiable stick and 
rudder feel to the cockpit as well as providing flight control inputs from the 
pilot to the simulation. In real-time, several characteristics of the stick and 
rudder pedals can be manipulated, such as position, force reflection, friction, 
damping, dead-band, velocity limits. Break points for non-linear position-based 
characteristics can also be manipulated. This capability provides the ability to 
modify stick feel as a function of any combination of variables within the flight 
simulation. The resultant algorithms could modify stick feel as a function of 
aircraft energy state or even tactical situation. 

5 Summary 

The human is a complex and adaptive receiver whose perceptual performance 
within a virtual environment is tied to the quality of the visual, auditory, and 
haptic stimulus generated within the multi-sensory virtual-interface. Many 
questions remain unanswered regarding virtual techniques, which, when an- 
swered, may increase the potential usefulness of virtual interfaces. In-context 
rapid prototyping is required to support creation and evaluation of multi-sensory 
virtual-interface concepts. Facilities can be developed utilizing off-the- shelf 
technology which provide high-fidelity flight simulation capability for in-context 
development of multi- sensory virtual-interfaces concepts for tactical aircraft. 
The Armstrong Laboratory has developed the FITE laboratory as part of an 
exploratory development program investigating multi-sensory virtual interfaces 
for tactical aircraft and is currently utilizing the facility to develop the initial 
multi-sensory virtually-augmented cockpit for air-to-air combat. 
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